Skip to content

fix(engine): skip video frame injection when a sub-composition host is hidden#1028

Open
lirian-su-opus wants to merge 5 commits into
heygen-com:mainfrom
lirian-su-opus:fix/inject-respect-ancestor-visibility
Open

fix(engine): skip video frame injection when a sub-composition host is hidden#1028
lirian-su-opus wants to merge 5 commits into
heygen-com:mainfrom
lirian-su-opus:fix/inject-respect-ancestor-visibility

Conversation

@lirian-su-opus
Copy link
Copy Markdown

@lirian-su-opus lirian-su-opus commented May 22, 2026

What

In packages/engine/src/services/screenshotService.ts + videoFrameInjector.ts, stop painting the page-side replacement <img class="__render_frame__"> when a <video data-start>'s visual ancestor is hidden, and make the skip robust against the two ways it could still cause stale or blank frames:

  1. Walk ancestors in both injectVideoFramesBatch and syncVideoFrameVisibility. display: none on any ancestor is always a skip signal; visibility: hidden is a skip signal only on sub-composition hosts ([data-composition-src] / [data-composition-file]).
  2. Hide stale frames with !important so applyDomLayerMask's #${showId} *{visibility:visible !important} rule cannot revive them when a sub-comp host lands in an active layer's show set.
  3. Return injected ids from injectVideoFramesBatch (Promise<string[]>) so createVideoFrameInjector only caches videos the page actually painted — skipped ones must re-attempt next call, not be remembered as up-to-date.

Add regression suite covering all three guarantees, plus the carve-out for plain [data-start] hosts whose visibility: hidden is escapable by the descendant <img>'s explicit visibility: visible.

Why

The original symptom — inner-PIP overpaint

The injector iterates every video[data-start] whose raw [data-start, data-start + data-duration) window covers the current seek and overlays the extracted source-video frame on each one. That contract is wrong for <video> elements nested inside data-composition-src sub-compositions:

  1. compileTimingAttrs auto-injects data-start="0" + a data-hf-auto-start marker on any <video> without timing attrs. The duration prober then resolves data-end to the full source duration. So an inner PIP <video> covers the entire timeline in the active check, regardless of which moment its host actually belongs to.
  2. The runtime's [data-start] lifecycle hides out-of-window hosts with visibility: hidden. The cascade hides the nested <video>'s rendered output — but the injector still creates an <img> next to it and explicitly sets img.style.visibility = "visible", which under CSS rules defeats the parent visibility: hidden.
  3. GSAP has not yet morphed the host (it advances only when the host enters its time window), so the <video>'s used box is the CSS default — typically full-bleed.

Net effect: every inactive sub-comp with a nested <video> painted one full-bleed replacement frame on top of whichever moment was currently visible.

Why the narrowing — style-9-prod

CSS-spec-wise, a descendant visibility: visible overrides an ancestor visibility: hidden. Regular [data-start] containers rely on that override: in style-9-prod, #aroll-comp has data-duration="16.04" but its GSAP timeline only tweens to t=9.88s; the runtime truncates the host to visibility: hidden after t=9.88s, and the replacement <img> must paint through to hold the final-state frame. An unconditional skip dropped 38 tail frames there and crashed PSNR from ~50 to ~11. display: none does not have this carve-out — it removes the subtree from layout entirely and no child override can escape.

Why mask defence

applyDomLayerMask writes body *{visibility:hidden !important} plus #${showId} *{visibility:visible !important} for each show id. If a sub-comp host's id lands in the show set, the show rule cascades to descendant __render_frame__ siblings. A plain inline visibility: hidden we wrote in the ancestor-hidden branch loses to important stylesheet author per CSS cascade — the stale frame would re-paint on the layer composite. Inline !important beats stylesheet !important (author-origin precedence), which is what applyDomLayerMask's own hide arm already does for extraHideIds.

Why cache hygiene

createVideoFrameInjector recorded lastInjectedFrameByVideo.set(id, frameIndex) after every call to injectVideoFramesBatch, including videos the page silently skipped. On the next call at the same source frameIndex — common when source fps < output fps, source is paused, or host start isn't frame-aligned — the cache short-circuited the second inject and the host's first visible frame painted blank because the <img> was never created. Returning the actually-painted ids and caching only those keeps the skipped videos re-eligible for injection on the next tick.

How

Ancestor check (both functions)

const isVisualAncestorHidden = (el: HTMLElement): boolean => {
  let parent = el.parentElement;
  while (parent !== null && parent !== document.documentElement) {
    const computed = window.getComputedStyle(parent);
    if (computed.display === "none") return true;
    if (
      computed.visibility === "hidden" &&
      (parent.hasAttribute("data-composition-src") ||
        parent.hasAttribute("data-composition-file"))
    ) {
      return true;
    }
    parent = parent.parentElement;
  }
  return false;
};

Walk starts at parentElement because both functions write visibility: hidden !important to the <video> itself, so reading the video's own computed visibility is unreliable.

  • injectVideoFramesBatch — short-circuit per video: hide a stale __render_frame__ sibling via setProperty("visibility", "hidden", "important"), then continue. Do not push the videoId into injectedIds. Return injectedIds to the caller at the end.
  • syncVideoFrameVisibility — fall through into the inactive arm so the <img> lands on inline !important hidden, matching what CSS already implies for the rest of the subtree.

The helper is inlined in both functions instead of hoisted to a shared module because the bodies run inside page.evaluate and a cross-file string-serialized helper would be more brittle than two identical copies. The regression tests exercise both paths so drift would surface.

Caller cache

const injectedIds = new Set(
  await injectVideoFramesBatch(
    page,
    updates.map((u) => ({ videoId: u.videoId, dataUri: u.dataUri })),
  ),
);
for (const update of updates) {
  if (injectedIds.has(update.videoId)) {
    lastInjectedFrameByVideo.set(update.videoId, update.frameIndex);
  }
}

Alternatives considered and rejected

  • Filter the active list at the call site (snapshot.ts, producer/services/videoFrameInjector.ts). Two equivalent implementations would have to stay in sync, and any future caller would have to remember to filter.
  • Read the host's data-start/data-end and recompute active per-video. Couples the injector to the timing convention; doesn't generalize to hosts hidden for other authoring reasons.
  • Have the runtime tear down inner <video> elements when the host is out of window. Same outcome but a much bigger blast radius — the injector is the layer that knows it's about to override the cascade, so it's the natural enforcement point.
  • Page-side roundtrip just to filter ancestor-hidden videos before the inject call. One extra page.evaluate per tick when injectVideoFramesBatch already has the DOM in scope and can return the same information for free.

Test plan

screenshotService.test.tsvideo-frame injection respects ancestor visibility (8 cases)

  1. visibility: hidden sub-comp host → injectVideoFramesBatch creates no <img> next to the video.
  2. display: none sub-comp host → same.
  3. Stale __render_frame__ <img> + sub-comp host now visibility: hiddeninjectVideoFramesBatch flips it to visibility: hidden.
  4. syncVideoFrameVisibility called with the video in active but a sub-comp ancestor hidden → the existing <img> stays hidden, not "visible".
  5. style-9-prod regression guard: plain [data-start] host at visibility: hiddeninjectVideoFramesBatch still injects the <img> with visibility: visible.
  6. style-9-prod regression guard (sync arm): plain [data-start] host at visibility: hidden, video active → syncVideoFrameVisibility flips an existing <img> to visibility: visible.
  7. Mask defence (inject arm): ancestor-hidden + pre-existing visible <img>injectVideoFramesBatch calls style.setProperty("visibility", "hidden", "important") so applyDomLayerMask's important stylesheet rule cannot revive it. Asserted via spy on the live <img>'s setProperty (linkedom strips !important from cssText/getPropertyPriority).
  8. Mask defence (sync arm): same setup, syncVideoFrameVisibility → same spy assertion.

setupHostHiddenScenario takes an optional hostAttribute so cases 5/6 swap the host's data-composition-src for data-start.

videoFrameInjector.test.tscreateVideoFrameInjector cache hygiene against page-side skips (2 cases)

  1. Cache not poisoned by silent page-side skip: first call returns [] (ancestor-hidden), second call at the same frameIndex must still issue an inject.
  2. Happy-path cache hit still works: first call returns ["pip"], second call at the same frameIndex short-circuits (no second inject). Pin so a future refactor can't trade the skip bug for a never-cache regression.

Stubs are wired via vi.mock("./screenshotService.js", ...) with hoisted spies on injectVideoFramesBatch and syncVideoFrameVisibility; the FrameLookupTable is faked with a one-payload getActiveFramePayloads, and frameSrcResolver short-circuits the on-disk cache.

Suite totals

bun run --filter @hyperframes/engine test: 619/622 passed. The three failures are in ffprobe.test.ts (HDR PNG cICP metadata parsing + ffprobe-missing fallback) and reproduce identically on main — unrelated to this change.

  • Unit tests added/updated
  • Manual testing performed (reproduced the symptom in a real hyperframes render run, confirmed the visible host's content shows through after the fix; verified bunx oxlint / bunx oxfmt --check and the engine test suite minus the pre-existing failures)
  • Documentation updated (n/a — fix is internal to the page-side injector + caller and does not change any external contract)

Copy link
Copy Markdown
Collaborator

@miguel-heygen miguel-heygen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style-9-prod regression — root cause found

The isVisualAncestorHidden check is too broad. It catches a legitimate rendering scenario in style-9-prod where the runtime's [data-start] lifecycle hides a composition container whose GSAP timeline is shorter than its authored data-duration.

What happens

In style-9-prod, the #aroll-comp element:

  • Has data-composition-id="aroll-layer", data-start="0", data-duration="16.04"
  • But its GSAP timeline only tweens up to 9.88s
  • The runtime truncates visible duration to Math.min(16.04, 9.88) = 9.88, setting visibility: hidden after t=9.88s

Before this PR: The replacement <img> had visibility: visible which correctly overrides ancestor visibility: hidden (CSS spec: child visibility: visible overrides parent visibility: hidden). The video held its final GSAP state.

After this PR: The ancestor walk finds #aroll-comp with visibility: hidden and skips injection entirely → 38 blank frames from t=9.95s to end (PSNR drops from ~50 to ~11).

Fix

The check should only trigger on sub-composition hosts ([data-composition-src] or [data-composition-file]) — which was the original intent of the PR. Regular [data-start] containers with visibility: hidden are handled correctly by CSS visibility inheritance (the <img>'s explicit visibility: visible overrides the ancestor).

Replace:

const isVisualAncestorHidden = (el: HTMLElement): boolean => {
  let parent = el.parentElement;
  while (parent !== null && parent !== document.documentElement) {
    const computed = window.getComputedStyle(parent);
    if (computed.display === "none" || computed.visibility === "hidden") return true;
    parent = parent.parentElement;
  }
  return false;
};

With:

const isVisualAncestorHidden = (el: HTMLElement): boolean => {
  let parent = el.parentElement;
  while (parent !== null && parent !== document.documentElement) {
    const computed = window.getComputedStyle(parent);
    if (computed.display === "none") return true;
    // Only treat visibility:hidden as a skip signal on sub-composition
    // hosts. For regular [data-start] containers the replacement <img>'s
    // explicit `visibility: visible` correctly overrides the ancestor per
    // CSS spec — we must NOT skip injection for those.
    if (
      computed.visibility === "hidden" &&
      (parent.hasAttribute("data-composition-src") || parent.hasAttribute("data-composition-file"))
    ) return true;
    parent = parent.parentElement;
  }
  return false;
};

Both copies of isVisualAncestorHidden (in injectVideoFramesBatch and syncVideoFrameVisibility) need this same change.

@lirian-su-opus lirian-su-opus changed the title fix(engine): skip video frame injection when a visual ancestor is hidden fix(engine): skip video frame injection when a sub-composition host is hidden May 25, 2026
LKI and others added 5 commits May 25, 2026 13:06
`injectVideoFramesBatch` and `syncVideoFrameVisibility` iterate every
`video[data-start]` whose raw time window covers the current seek.
Inner `<video>` elements inside `[data-composition-src]`
sub-compositions get `data-start="0"` auto-injected by
`compileTimingAttrs` and probed-duration cover the entire timeline,
so they look "active" even when their host has not yet started.

When the runtime then hides the host with `visibility: hidden` (its
out-of-window lifecycle), the inner video inherits hidden via the CSS
cascade — but our injector responded by painting a replacement
`<img class="__render_frame__" style="visibility: visible">` next to
the video. `visibility: visible` on the descendant defeats the parent
`visibility: hidden` cascade, and because the host has not been
morphed by GSAP yet the video's bounding box is its CSS default
(usually full-bleed). The result is one full-bleed frame per inactive
sub-comp painted over whichever moment is *actually* visible — the
overlay symptom the upstream agentic-finecut project saw.

Walk ancestors in both functions; if any has `display: none` or
`visibility: hidden`, skip the inject and hide any stale
`__render_frame__` sibling. The render is now correctly empty for
hidden hosts, which is what the surrounding CSS cascade already
intends.

Tests:
- `screenshotService.test.ts`: cover the new guard for both
  visibility:hidden and display:none hosts, both for the fresh-img and
  the stale-img paths, plus `syncVideoFrameVisibility` for the case
  where the time window calls a video "active" but a hidden ancestor
  still requires its frame to stay hidden. Each test fails against
  pre-fix `screenshotService.ts`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The screenshotService.test.ts regression-suite comment pointed at the
author's fork branch as backstory. Strip the line so upstream code
doesn't carry a fork-relative reference; the surrounding paragraph
already explains the bug end-to-end without it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`isVisualAncestorHidden` was treating any `visibility: hidden` ancestor as a
signal to skip injecting the replacement frame. That's too broad — for plain
`[data-start]` containers, the replacement `<img>`'s explicit
`visibility: visible` correctly overrides the ancestor per CSS spec, and
consumers rely on that to hold the final GSAP-driven frame when an authored
`data-duration` outlives the composition's GSAP timeline (e.g.
`style-9-prod`, where the runtime truncates the host to `visibility: hidden`
after the timeline ends and the replacement frame must paint through).

Restrict the `visibility: hidden` skip to ancestors that carry
`data-composition-src` or `data-composition-file` — the actual sub-composition
hosts this guard was added for. `display: none` keeps the broad behavior:
it takes the whole subtree out of layout and a child override cannot escape.

Update the existing regression suite to mark the host as a sub-composition,
and add two new cases pinning the plain-`[data-start]` behavior: both
`injectVideoFramesBatch` and `syncVideoFrameVisibility` must still produce a
visible replacement `<img>` when the host is `visibility: hidden` but does
not carry a sub-composition attribute.
…cache

Two follow-ups to the ancestor-visibility skip in `injectVideoFramesBatch`
and `syncVideoFrameVisibility`.

1. **Mask defence.** Both ancestor-hidden branches previously wrote a plain
   `img.style.visibility = "hidden"`. `applyDomLayerMask` writes the
   stylesheet rule `#${showId} *{visibility:visible !important}`, and CSS
   cascade puts important stylesheet author above non-important inline
   author — so a sub-comp host landing in the active layer's `show` set
   would revive a stale `__render_frame__` and let it bleed onto the
   layer composite. Write the hide via
   `style.setProperty("visibility", "hidden", "important")` instead;
   important inline beats important stylesheet.

2. **Caller cache hygiene.** `createVideoFrameInjector` unconditionally
   wrote `lastInjectedFrameByVideo.set(id, frameIndex)` after calling
   `injectVideoFramesBatch`, even for videos the page silently skipped due
   to a hidden visual ancestor. On the next call at the same frameIndex —
   common with source-fps < output-fps, paused source frames, or
   non-frame-aligned host starts — the cache short-circuited the second
   inject and the host's first visible frame painted blank because the
   replacement `<img>` was never created.

   Make `injectVideoFramesBatch` return `string[]` (the subset of ids it
   actually painted) and have the caller cache only those. The cli-side
   `snapshot.ts` consumer is unaffected: its local `InjectFn` types the
   return as `Promise<void>`, which is structurally compatible with
   `Promise<string[]>` under TS void-return assignment rules.

Tests: linkedom doesn't preserve `!important` in cssText, so the two new
mask-defence cases spy on the live `<img>`'s `style.setProperty` and assert
the 3-arg call shape. The cache-hygiene case stubs the page-side primitives
via `vi.mock`, drives the hook twice at the same frameIndex with a stubbed
"injected nothing" first response, and verifies the second call still
issues an inject. A counter-test pins the happy-path cache hit so a future
refactor can't trade the skip bug for a never-cache regression.
`injectVideoFramesBatch` now returns `Promise<string[]>` so the caller can
filter cache entries to videos the page actually painted. The cli-side
snapshot command does not use the return value, but its local `InjectFn`
declared `Promise<void>` which made the `as { injectVideoFramesBatch:
InjectFn }` cast on the dynamic engine import fail typecheck under TS's
"sufficiently overlapping types" rule. Match the engine's actual export
shape.
@lirian-su-opus lirian-su-opus force-pushed the fix/inject-respect-ancestor-visibility branch from aae9454 to dcd637b Compare May 25, 2026 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants